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ABSTRACT 

The evaluation of a school improvement program for 
elementary schools (School Effectiveness Training) and for secondary 
schools (Secondary School Development Program) provided an 
opportunity to determine if user participation could increase 
knowledge use. It was hypothesized that program staff participation 
might be a solution to the dilemma of knowledge use. Interactive 
techniques were employed to involve program staff. These included 
joint site visits, debriefing sessions, informal interaction, 
feedback meetings, field notes, written statements, and formal 
reports. The use of evaluation information was accelerated as a 
result of program staff participation. The risks which may accompany 
staff participat ior had only slight negative impacts in this study. 
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The way program evaluation findings are used has been a recurrent 
theme in evaluation literature for nearly two decades. Once, evaluators 
expected their studies to provide a significant portion of the knowledge 
base that policy makers consider when making important decisions about a 
program. Yet as the evaluation field has matured, members have become 
more sophisticated about how much evaluation, in fact, is used in 
decision making. They have come to see that many factors other than 
formal inquiry (for example, practical and political considerations or 
common sense knowledge) often strongly influence programs* directions 
(Llndblom & Cohen, 1979; Weiss, 1980). Evaluators also have come to 
recognize that '*use'* can be a relative concept and that different levels 
of information use occur, from altering perceptions of a program to 
influencing major decisions about it (Alkin, Daillak, & White, 1979). 
Lastly, they have realized that it is possible to enhance knowledge use a 
number of ways and that one effective way to do this is by deliberately 
involving potential information users such as clients, sponsors, and 
other audiences in their evaluation studies (for example. Gold, 1983; 
Stake, 1975). 

In what follows, we will take a closer look at how one evaluation 
study tried to meet the knowledge use issue head on by involving program 
staff in it. First, we will outline some hypotheses about participation 
as a potential solution to the dilemma of knowledge use.. Then we will 
describe an evaluation strategy that incorporates user participation. 
Finally, we will draw some conclusions about the effectiveness of this 



strategy for increasing the use of this study's information as well as 
about the implications of employing this kind of evaluation strategy in 
other instances. 

Using Knowledge 

As Ernest House (1973) noted, "Producing data is one thing: getting 
it used is quite another" (p. 133). This sentiment describes a major, 
chronic dilemma of evaluators. As far back as the middle sixties both 
Cuba (1968) and Stufflebeam (1967) noted that evaluation information is 
often seen as useless and irrelevant to decision makers. Since then 
knowledge use — or non use — has been documented quite diligently by 
evaluators and policy researchers, particularly more recently (for 
example, see Alkin, Daillak, & White, 1979; Deshpande, & Zaltman, 1983; 
Florio, Behrraan, & Goltz, 1979; Leviton & Hughes, 1981; Weiss & 
Bucuvalas, 1980). This attention has improved our understanding of the 
problem of information non use, its causes, and how it might be overcome. 
One way to increase knowledge use suggested by a number of authors is by 
involving clients or others in evaluation activities. 

Knowledge Use and Client Involvement 

In examining the impact of health evaluation research, Pat ton and 
his colleagues (Patton, Grimes, Guthrie, Brennan, French, & Blyth, 1975) 
discovered that evaluation information probably does get used by decision 
makers. However, they suggested that it is only one of a number of 
pieces of information that influence decisions and it often competes with 
other considerations, which Patton and his colleagues label "political" 
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and "personal." These authors see the personal considerations as the 

most interesting for they seem to be more susceptible to evaluator 

influence. Also, they might be influenced by increasing client or 

sponsor participation. These personal considerations are: 

o general lack of enthusiasm by the client for using information 
generated by evaluation studies, 

o low regard by clients of the evaluator or evaluation design, and 

o low commitment by the client to using evaluation information in 
general. 

While reviewing the research of Patton and his colleagues as well as many 
others, Leviton and Hughes (1981) zeroed in on these personal consid- 
erations and described clusters of variables associated with personal 
considerations that they felt would enhance a client's use of 
information : 



o relevance of the information for program concerns, which includes 
how well it addresses the needs and concerns of managers and 
developers, as well as the timeliness of the information; 

o effectiveness of communication, which in the main refers to 

direct communication of information to users without intervening 
layers of bureaucracy; 

o ease with which the clients process the information, which 
includes how well they recognize its relevance, how quickly 
findings are turned into implications, how the information is 
presented, and the information processing skills and styles of 
the key actors; 

o credibility of the information, which includes how it fits with 
other information or preconceptions that potential users have, 
the quality of the evaluation, and user opinion of the evaluator; 
and 

o involvement and advocacy for using the information on the part of 
a key actor: client, manager, or developer. 

They concluded that client involvement in research studies was an 



effective way to give these variables a positive valence and thereby 



enhance the use of evaluation information. They argued that by 
Involving clients in study activities, evaluators stood a better chance 
of relating research information to user needs and concerns, making sure 
the information is communicated smoothly and efficiently, helping users 
recognize its usefulness, keeping their own and the evaluation's 
credibility high, and gaining the advocacy and involvement of a key 
actor . 

Client involvement in evaluation and policy research studies is not 
a unique idea. Two evaluation approaches described by Stake (1973) and 
Gold (1983) explicitly call for user involvement at several stages of the 
evaluation process. Stake's approach, which he labels "responsive" 
evaluation, calls for the evaluator to consult users and try to 
incorporate their interests and values while the study is being designed. 
He also recommends that their reactions to report drafts be solicited 
and, whenever possible, they be consulted in the interim. Gold goes even 
farther in his "stakeholder" approach. According to him the evaluator 
should adhere to user (stakeholder) preferences — in effect, work for the 
stakeholders, who specify what kinds of information they want and in what 
form. Client participation also has been advocated by Ballard and James 
(1983) and Leitko and Peterson (1982). The -former suggest several 
participation strategies including using advisory committees, circulating 
report drafts, interacting verbally and in writ.ing, and disseminating 
research products. Leitko and Peterson go beyond Ballard and James by 
advocating that participants be involved in research decision making. 

Given the findings and recommendations of these authors, it seems 
that involving potential users in evaluation activities might be an 
effective way to overcome some barriers to knowledge use. First, it 



might help clients to understand research activities more fully than if 
they were not involved. Second, it might make them more aware of the 
relevance, credibility, and value of the evaluation information 
—particularly if they helped generate it. Third, it might make it 
easier for them to communicate with evaluators. Fourth, it might cause 
them to become advocates of the evaluation process. Fifth, and most 
important, in might enable the clients to process the information more 
quickly and easily and use it more effectively in decision making. 

User Participation in an Evaluation Study 

During the past two and one-half years, we have had an opportunity 
to see if user participation could increase knowledge use. We are 
evaluating two programs chat staff members of Research for Better 
Schools, Inc. (RBS [also our employer]) are conducting in schools. Staff 
members who are responsible for those programs have been extensively 
involved in the evaluation study. 
The SET and SSDP Programs 

In 1980 staff of the Urban Development (UD) Component at RBS began 
work on a school improvement program for elementary schools. School 
Effectiveness Training (SET); in 1983 a program for high schools, the 
Secondary School Development Program (SSDP), was added. The two programs 
are similar in that both are approaches to improving urban schools. 
Moreover, both are sponsored and carried out as a joint effort of the 
state and local education associations, RBS, the local school district, 
and the school building staff. The rationale underlying both is that 
many of the problems of schools are problems of organization, motivation. 
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connnunicacion, coordination, and resource allocation. That is. they are 
work place problems similar to those in other work settings and 
theretore, can be solved by organizational development strategies. 

Both programs stress the involvement of the total school comunity 

c ui» orpac anA in the improvement of the school's 
in the definition of problem areas and in cne j-m^ 

f£ectiveness. And both are based on the prer.ise that schools can become 
re effective by making use of existing resources, by opening up the 
decision-making process, and by making use of the influence of people who 
carry out the decisions in the daily work of the school. 

The only major differences between SET and SSDP arise out of the 
differences inherent between elementary schools and high schools: size, 
complexity, diversity, department structures, special programs, and the 
like. 

Evaluation of SSDP and SET 

Both programs include an evaluation component which was added in 
response to stipulations of the funding agent, (the National Institute of 
Education, [NIE]), to RBS ' corporate policies, and to the recognition by 
the UD developers that knowledge about program implementation and 
effectiveness is vital for program improvement. Basically, the 
evaluation consists of a documentation study of the use of SET and SSDP 
in schools. 

The study design emphasizes utility and efficiency. The evaluators 
and developers both strongly believe that the most important function the 
study serves is to provide information that program staff can use to help 
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Improve the programs in each site and refine them as they are introduced 
in new schools. Because the study's resources were quite limited, the 
methodology also had to be tailored with economy in mind. 

The documentation study focuses on these six areas of information 
earmarked by both developers and evaluators as especially relevant to the 
success of the two programs: 

o program initiation at each site, 

o program structure and operation at each site, 

o support for the program among school staffs, 

o amount and type of staff participation in the program, 

o characteristics of program processes and structures at 
each site, and 

o changes introduced in schools because of the program. 

The methodology chosen is largely qualitative and informal; it is 
designed to collect information as completely and economically as 
possible, while providing timely feedback to program staff and 
maintaining the quality of information through the use of multiple data- 
collection methods. Most information is collected through 
semi-structured interviewing and observation. Other techniques for 
obtaining information include reviewing documents and administering 
questionnaires. 

Staff Participation in the Evaluation 

A major feature of this evaluation design is that program staff 
members are extensively involved in the study. They are RBS employees 
who have helped design the SET and SSDP Programs, who provide training 
for teachers » and who continue to assist teachers and program 
implementation. Some have participated actively, helping to conduct 
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interviews or even becoming full partners and contributing to all aspects 
of the study. Others have been involved less directly, primarily through 
informal interaction with evaluators during feedback activities. These 
program staff became involved more out of economic necessity than 
design — the evaluator was assigned to the study only part-time (at first » 
one-quarter time, now one-half time). Study resources were to be 
increased by enlisting staff assistance. As it turned out, however, 
their participation enhanced the study and the use of study results (both 
of which will be discussed in a forthcoming section). 

During most of the two and one-half years of the SET/SSDP 
evaluation, one or two v^ogram staff members have been assigned to it as 
active, direct participants. Initially, two field agents, whose primary 
program roles were to help develop materials and provide technical 
assistance to school- level participants, helped with data collection. 
They did this during the study's first year. They accompanied the 
evaluator to sites, interviewed teachers, and prepared written field 
notes. Later, another staff member (one of the co-authors) became a full 
participant in the study, contributing to design, data collection, and 
feedback activities. 

Beyond this direct participation, other program staff 
members — including the director—have been involved deliberately and 
extensively in the evaluation study. Most of this involvement is in 
activities related to data collection or feedback. These activities 
involve staff in the evaluation study in the sense that they stimulate 
and encourage two-way communication between the evaluators and the staff 
members. The involvement techniques are: 
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Joint site visitc. Program staff and evaluators often 
attend onsite program development sessions or other 
project meetings together. 

Debriefing sessions. Evaluators ask staff members to 
describe events the evaluators do not attend as veil as 
other personal or telephone contacts with sites. 

informal interaction. After collecting Information about 
a particular site, evaluators usually give Informal 
feedback to the staff members responsible for program 
operations In that site. 

Feedback meetings. Evaluation findings and their program 
implications are discussed during formal meetings convened 
specifically to present recent informmtion. Findings also 
are discussed during routine staff or other meetings. 

Field notes. Field notes, which have been edited to 
protect Informant anonymity (as well as relationships 
between staff and evaluators), arc sometimes shared with 
individual staff members. 

Written statements. Occasionally, after completing a 
round of interviews or analyzing questionnaire results, 
evaluators prepare written memos presenting tentative 
findings. 

Fomal reports. Formal evaluation reports are written 
annually and distributed to staff members in draft form. 

All of these techniques are interactive. In addition to being read, 

field notes are usually discussed informally; written statements are used 

as a basis for feedback meetings; in reacting to drafts of formal 

reports, program staff typically discuss study findings before they are 

finalized. 

Although a variety of techniques have been used, few are used 
frequently, minimizing the burdens on both evaluators and program staff. 
Most feedback is communicated informally through personal interaction. 
Other techniques have been routinized effectively or are used primarily 
when staff are making important decisions or when evaluators identify a 
need to share new data. The evaluators attempt to talk with program 



staff frequently and to provide feedback in whatever form seems most 
effective for communicating information for a particular purpose, but — as 
they do with all their interactions with program staff — they also try co 
keep their feedback brief and to the point. 

The Benefits of Staff Participation 

Parcicipation by program staff in evaluation has had benefits to the 
evaluation study, the program and its staff, and to program clients. It 
seems to be a particularly effective strategy for increasing information 
use. At the same time, it helps keep costs down and produces other 
benefits such as increasing the evaluation knowledge base, helping staff 
and evaluators develop shared understandings of the program studied, and 
facilitating program improvement. 
Increased Utilization 

The major benefit of program staff participation has been that it 
accelcrcnr.es the use of evaluation information. Staff members learn more 
about assistance needs in program sites as well as about program 
adjustments that will facilitate its introduction into new schools. 
Furthermore, this information is acquired sooner than in many evaluation 
and policy research studies. Program staff have used the data to alter 
and design new technical assistance activities for single and multiple 
sites and to make a number of strategic changes in the program. For 
example: 

o developing written materials and holding cross-site 
seminars to streamline the technical assistance 
process, f^-pecially as it is needed in multiple sites; 

o reducing emphasis on specific types of changes in order 
to enable participants in sites to broaden and increase 
the flexibility of the issues or problems they address; 
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o building inco che program provisions for technical 

assistance Co principals and for deliberacely eliciting 
explicit district commitment to che program; 

o modifying efforts to stimulate high enthusiasm among 
participants In order to prevent severe declines In 
motivation and perceptions that expectations were 
raised falsely; 

o revising future technical assistance plans, primarily 
to increase that assistance; 

o ceasing to initiate programs in new sites late in the 
school year so that activities can be well underway 
before the year ends and initial momentum is lost; 

o advising central councils to adopt short-term as well 
as long-term projects in an effort to ensure that 
visible progress occurs without long delay; 

o making copies of school improvement plans developed 
during initial sessions available to schools quickly 
to enable school staff to use them for development; and 

o responding to crises in individual sites. 

This increased use of evaluation information can be explained by 
referring to Leviton and Hughes' clusters of variables which affect 
utilization, as outlined earlier. Specifically, staff involvement has 
increased relevance by influencing the types of data sought. Program 
staff members who participate actively make suggestions about the topics 
which will be covered during interviews; furthermore, during Interviews 
thev sometimes seek answers to questions of their own. Other staff 
members have increased relevance by informally telling evaluators of 
their current activities, concerns, and information needs. 

Staff participation also has resulted in improved communication, 
which is perhaps the major reason utilization has increased. 
Communication is direct, informal, interactive, frequent, and timely. 
Program staff who interview teachers directly hear their versions of 
program activities as well as problems they think have hindered progress. 

O 11 
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This gives program staff a more vivid understanding of the program's 
actual use than they could have obtained from reading an evaluation 
report. livalu;ic Ion Information is communicated to other program staff 
members less directly, but it stilJ goes through only the evaluators and 
typically is communicated to staff directly without being translated into 
formal written reports. The knowledge is usually communicated informally 
and interactively, giving program staff an opportunity to ask clarifying 
questions and discuss findings among themselves and with the evaluators. 
The involvement of program staff also means that communication occurs 
more frequently and is more timely; evaluators tell staff members of 
assistance needs soon after identifying them so that assistance can be 
provided without the delay that would be likely otherwise. 

Leviton and Hughes' cluster of variables labeled "information 
processing** also helps explain the increased use of evaluation 
information that comes from the involvement of program staff. Because 
they help collect findings, users are aware of their relevance. The 
implications for action are usually obvious, although group feedback 
sessions help translate findings into policy changes. Most of the 
information is qualitative, which Leviton and Hughes say users prefer 
over quantitative data. And differences in the information-processing 
styles of evaluators and audience members — another important 
consideration in this cluster— have been minimized as the latter become 
participants in the evaluation study. 

Credibility, another Leviton and Hughes* cluster of variables, is 
enhanced by program staff members' familiarity with the documentation 
study and by the frequent interaction among evaluators and program staff. 
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This interaction reassures program staff that the evaluators are not "out 
to get" the program, helping staff develop trust in the study. Those 
staff who are directly involved learn about the evaluation process; they 
then help convince others of the study's credibility. In particular, the 
program developer who became an evaluator has helped increase credibility 
because of his close long-standing relationship with program staff 
members, who see him more as one of them. His reports and 
interpretations of interviewees* comments carry a little more weight, and 
his increased knowledge of program development enables him to suggest 
strategies for overcoming problems the study identifies. 

Another cluster of Leviton and Hughes* variables refers to user 
commitment and advocacy. This too has been enhanced by involving program 
staff. Their commitment to the evaluation study has increased as they 
see that it produces useful information. Staff who participated directly 
have become advocates of the study. They encourage evaluators to 
interview school administrators and teachers, go to the evaluators with 
questions, and request feedback sessions prior to new program 
initiatives. The program director has been a study advocate from the 
beginning, helping ensure that findings would be used. 
Other Benefits 

Another important benefit of staff participation has been that it 
has increased the study's knowledge base without the expense and effort 
associated with hiring someone and orienting him or her to the study and 
program. Furthermore, the backgrounds and perspectives of the three 
program staff members who participated directly enhance their 
contributions to the study. Two had been involved in many previous 
similar efforts to develop and implement new programs in schools. They 
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understand the process of change and have been quick to identify factors 
that facilitate or hinder program-related changes in individual sites. 
The other had helped develop program procedures and materials, was very 
familiar with the program, and knew what questions to ask about it. The 
less direct involvement strategies used with other staff have also 
enlarged the data base, primarily by adding information and alternative 
viewpoints. A continuing benefit is that program staff provide 
information voluntarily; having become accustomed to frequent interaction 
with the evaluators, staff feel comfortable approaching them to discuss 
new activities and implications for program sites. Program staff treat 
the evaluators as partners in a collegial relationship, particularly one 
evaluator who has worked with the staff for several years on a variety of 
projects and whose office is near theirs. 

A further benefit has been that shared understandings of the program 
have developed as staff members and evaluators have interacted with one 
another. Thus, many study conclusions have emerged gradually and formal 
reports are not received with shock or defensive reactions. In fact, the 
opposite has occurred. Program staff have supported most 
formally-reported conclusions, which they helped formulate over a period 
of time and could not very well dispute later. In contrast to other 
experiences, one evaluator found that the SET/SSDP program director 
challenged only one conclusion in a recent formal report — a positive one 
which he said had been over-stated. 

For program clients — the funding agent and the staff at the various 
sites — program staff involvement in evaluation has meant continuous, 
rapid, focused improvements in the structure and conduct of the programs, 
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Staff have provided assistance to sites more readily and rapidly than if 
they were not involved in the evaluation and had less access to 
information. Staff have also focused more on correcting and modifying 
weak aspects of the program. The net result is a stronger, more 
effective program and a more efficient delivery of that program. 

Literature on audience involvement and knowledge use suggests that 
other benefits may be occurring also. Leitko and Peterson (1982). for 
example, say that validity may be increased when the relationships 
between researchers and participants (in this case, program developers) 
improve. This occurs because participants become less threatened by 
the evaluation study, and thereby more motivated to participate and 
respond openly and honestly. Another suggested benefit of this kind of 
participation is that the evaluation information generated may have a 
lasting effect on program staff because it is more meaningful to them and 
becomes incorporated into their working knowledge (Kennedy, 1983). 

This is definitely true in the case of one of the co-authors who, in 
the course of carrying out his evaluation responsibilities, came to a 
greater understanding of the educational change process, what hinders or 
helps it, and how to capitalize on contextual, political, and other 
similar variables to introduce changes and make them stick. It seems 
likely that this happened to other program staff also — either through 
their direct participation or their vicarious involvement in feedback 
discussions, for example. 
Direct Versus Less Direct Involvement 

Direct involvement has tended to be more beneficial than less 
direct. The field agents who interviewed teachers acquired more 
information than did staff who were less directly involved, and they 
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acquired it more directly and immediately. In addition to spending 
several hours talking to teachers, the field agents read previous 
evaluation field notes in preparation for interviews and talked with the 
evaluator enroute to and from a site as well as later in the office, thus 
quickly learning more about the site's background and the perceptions of 
teachers the evaluator interviewed. Program staff who are less directly 
involved, even though their information is filtered through the 
evaluators and not for several days, still receive more information 
sooner and more directly than staff do in most evaluation studies. 
The Potential Risks of Staff Participation 

There are also a number of risks associated with using an evaluation 
strategy that involves program staff. In contrast to the benefits 
described above, the risks are relatively hypothetical. They are 
disadvantages which we think could have occurred but whose negative 
impacts were so slight that we prefer to label them "risks." One major 
risk is cooptation of evaluators and developers. Evaluators run the risk 
of avoiding threatening issues, equivocating negative reports, or 
generally being less than candid in order to protect their relationships 
with developer participants. Program staff, on the other hand, run the 
risk of relying too heavily on information they feel they have helped 
generate — for example, accepting and acting on evaluators^ 
recommendations at face value without critical analysis. To a minor 
degree some cooptation probably has occurred in this study, although it 
is difficult to point to specific instances. It is our impression, 
however, that there have been no serious consequences. 

Another related risk has to do with the integrity of the study. 
When evaluators emphasize collecting information that is useful to 
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ongoing program operations, they may neglect background knowledge which 
could become very important to understanding how a program evolves in a 
parcJcuIar school but whose immediate utility is not apparent. That has 
occurred some in this study, but it results as much from the lack of 
sufficient resources as from the study's focus. Furthermore, the 
evaluators are very aware of the importance of background knowledge and 
attempt to collect it whenever the opportunity arises. So, this is not a 
serious problem for us yet. 

Another more serious problem for a study's integrity may be that 
program staff who participate actively replace evaluation personnel. 
Program staff members are usually not trained in evaluation methods, even 
though they may be very well trained in their own areas of expertise* In 
addition, their status may be higher — or lower — than that of persons who 
would have been hired specifically for evaluation tasks. This may tend 
to influence the nature of evaluation activities. For example, some 
people are more adept than others at interviewing or at analyzing 
statistical data. Also, some are more willing to do the tedious tasks 
associated with compiling questionnaire results or other quantitative 
data. In this study, depth of staff skills in evaluation as well as 
their status did influence evaluation activities* Fortunately, the 
evaluation study was sufficiently flexible to adapt to the type of 
assistance available, primarily by reducing the emphasis on quantitative 
information—a direction that was compatible with the staff's information 
needs. Also helping to minimize this risk was one staff member's 
training in anthropology and his consequent understanding of the goals 
and techniques of qualitative methods of inquiry. 
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Another risk of involving program staff in evaluation activities is 
the imposition that it makes on their time and priorities. In this 
instance, staff involvment took time away from developing or assisting in 
the field in order to learn new skills or conduct evaluation activities. 
It also meant wearing two hats and, in some cases, reconciling their own 
value conflicts— whether to approach a situation as an evaluator or a 
developer or a field assistor. So far no one has complained too loudly 
about these impositions; continued program staff involvement and the 
extra responsibilities it brings with it may cause a change, however. 

Conclusion 

Having weighed the benefits and risks of involving program staff in 
evaluation studies, we believe that it has been a very worthwhile 
strategy so far for this study. Increasing the use of evaluation 
information has been a very important benefit which, in turn, has led to 
program improvement. Expanding the study's knowledge base at a 
relatively low cost has been particularly advantageous in this low-budget 
study. The development of shared understandings of the programs by 
evaluators and staff members have not only enriched the study, but staff 
have been more receptive to findings they helped generate and that 
emerged gradually. The risks that can accompany staff participation have 
had only slight negative impacts in this instance. Cooptation has been 
minimal, the study's integrity has not been compromised seriously, and 
the impositions on evaluators and staff have been far less than the 
benefits. 

We wish to emphasize, however, that a number of situational factors 
facilitated this involvement and worked to minimize the risks. The fact 
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